In general, prediction systems utilize information associated with agents in an environment to infer future actions of the agents, such as trajectories. Some existing prediction systems utilize hand-crafted or hand-coded rules to predict these future actions, for example. However, sensor data can be noisy and does not always correspond to such hand-crafted or hand-coded rules, introducing prediction errors and thus making resulting predictions inaccurate and/or unreliable.